Team JAMJAM#2293
Conversation
…ints/visualization/tests/ - Move flat files into subdirectories: - bbox_selection.py → modules/bbox_selection_module.py - bbox_distance_follow.py → modules/bbox_distance_behavior_module.py - go2_startup_self_check.py → modules/go2_startup_self_check_module.py - yoloe_tracking.py → modules/yoloe_tracking_module.py - test_bbox_distance_follow.py → tests/test_bbox_selection_module.py - Add blueprints/ with bbox_distance_follow, go2_startup_self_check, yoloe_tracking_test - Add visualization/detection2d_overlay.py (detections/selected_bbox/yoloe overlays) - Update all_blueprints.py with new import paths - Update README.md with new directory structure and RPC import examples
Compose YOLOE detection + keyboard-controlled Go2 in simulation or on real hardware. - Force YOLOE device=cpu in simulation to avoid CoreML/MPS conflict with MuJoCo locomotion policy (CoreMLExecutionProvider) - Route KeyboardTeleop through MovementManager (tele_cmd_vel) to match unitree_go2 control path; direct cmd_vel bypass broke MuJoCo sim - Rerun layout: 2D camera view with cyan YOLOE overlay + 3D view - Register as 'yoloe-keyboard-teleop' in all_blueprints.py
…h estimation - BBoxSelectionModule: add click_hit_padding_px and click_snap_distance_px so edge/small bbox clicks are more forgiving; nearest-bbox snap fallback for near-miss clicks - BBoxSelectionModule: downgrade per-click diagnostic lines to debug, keeping only key state-change lines at info - TargetLockModule: guard _on_detections publish paths against concurrent target-id changes (stale lock prevention) - BBoxDistanceBehaviorModule: add full task lifecycle logging (started / blocked / completed / ended / stopped) - BBoxDistanceBehaviorModule: suppress repeated 'task started' logs while already approaching same target; retain completed target id in done state to prevent same-target restart loops - BBoxDistanceBehaviorModule: adaptive depth estimation bbox expansion (depth_bbox_padding_px, depth_bbox_max_padding_px, depth_bbox_padding_step_px) to recover from sparse-pointcloud no-depth failures - websocket_server: downgrade null-field fallback and publish-detail logs to debug - tests: add regression tests for padding hit, nearest-snap, expanded-depth, and websocket null-z handling - README: document click-hit and depth-estimation tuning knobs with recommended adjustment order
…on3DPC world-frame distance The previous _estimate_bbox_distance() projected raw lidar points directly using camera intrinsics, treating world-frame XYZ as camera-frame XYZ. Go2 lidar is already in world frame (frame_id='world'), so the projected 'depth' values were meaningless — using world Z (up) instead of camera Z (forward), producing distances of 0.082–0.245m regardless of real range. Method B fix: - Use Detection3DPC.from_2d() with self.tf to transform world→camera_optical - Project world-frame lidar points into bbox, get world-frame 3D centroid - Compute 2D Euclidean distance between robot base_link and detection center - Keep existing angular P-controller (pixel-based horizontal offset) - Remove depth percentile / bbox padding config fields (no longer needed) - Add tf.start() call in start() to activate TF subscription - Add _safe_track_id() helper for non-numeric detection.id fields Also includes teleop interrupt (KeyboardTeleop + MovementManager wiring) added in the previous session: n_workers=12, remappings for nav_cmd_vel / tele_cmd_vel, stop_movement → teleop_active interrupt.
- Switch yoloe_target_lock_distance_follow blueprint base from unitree_go2_basic to unitree_go2, adding VoxelGridMapper, CostMapper, and ReplanningAStarPlanner for SLAM map rendering and click-to-navigate - Remove duplicate MovementManager.blueprint() (already in unitree_go2) - Remove world/lidar visible=False override so SLAM map appears in 3D view - BBoxDistanceBehaviorModule: add optional _planner: ReplanningAStarPlannerSpec injected by blueprint; call cancel_goal() when a new bbox task starts - MovementManager._on_click: publish stop_movement before forwarding goal, so any active bbox tracking task is cancelled when user clicks SLAM map - n_workers: 12 -> 16 to accommodate added navigation modules
What lands on jamjam_ui:
1. dimos/apps/marauders_map/ — new namespace, full Harry-Potter "Wrath of
Filch" web app on :7782. Composes upstream nav + jamjam's YOLO-E
target-lock/follow stack into one autoconnect() blueprint. Parchment
UI, intro overlay with magick-in char-by-char reveal, click-to-track
from map + roster + gallery, click-to-navigate, on-page WASD teleop,
quest chip, looped BGM.
2. dimos/robot/unitree/ hardware fixes:
- connection.py: free_walk() now also sends SwitchJoystick(1027,
data=True) — fixes the "dog sways in place" symptom where lx/ly
were interpreted as body lean instead of velocity.
- go2/connection.py: 2 FreeWalk publishes on init with a settle
gap + 15s background heartbeat. Guarded by
hasattr(connection,"free_walk") so MuJoCo/DimSim/Replay don't crash.
3. scripts/run-blueprint.sh — auto-logged runner. Mode resolution by env
(SIM > REPLAY > ROBOT_IP). Uses mjpython on macOS for MuJoCo's
main-thread GL context. Writes /tmp/dimos-logs/<bp>-<ts>.log with
git HEAD + uncommitted file count header. Auto-rotates to newest 20.
4. examples/go2_phone_control/ — phone-as-controller webapp (mock + real),
plus minimal direct-WebRTC scripts (go2_walk_*.py, go2_person_aware_walk.py)
and a standalone halt.py for E-STOP.
5. HACKATHON.md — branch-level README with the structure, run modes,
data flow diagram, and E-STOP layers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add four Hogwarts house badges (Gryffindor / Hufflepuff / Ravenclaw /
Slytherin) and serve them via /icons/{house}.png from ReidMapModule.
* Assign every HP character in hp_characters.js a canonical `house`
field. Teachers, non-Hogwarts wizards, creatures, and Muggles get
null = no badge (per design rule).
* Render the badge in three places in marauders_map.html: roster row
inline, gallery card top-right, hover tooltip dedicated row.
* Clicking a person in the right-side "Those Present" list now opens
the same HP-flavoured confirm parchment that a footprint click on
the map opens, instead of selecting silently. Tracked-person
re-click still releases without re-prompting.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Greptile SummaryThis PR adds Team JAMJAM's hackathon focus-helper application: a Harry-Potter-themed "Marauder's Map" web UI that plots detected people on an occupancy-grid floor plan and lets the robot dog follow a selected person (to keep a phone out of reach). The stack builds on a new
Confidence Score: 3/5The web-UI stop signal won't reach the selection/lock modules due to a missing topic remapping, and the connect-event handler iterates a dict that the detection thread can concurrently mutate — both need fixes before the Marauder's Map blueprint is reliable on real hardware. Two issues need attention before this runs cleanly in the field. The
|
| Filename | Overview |
|---|---|
| dimos/apps/marauders_map/module.py | New 835-line ReidMapModule: Socket.IO web server serving the Marauder's Map UI; has a thread-safety bug where the connect handler iterates _people from the asyncio thread while the detection stream thread can concurrently modify it, and CORS is fully open. |
| dimos/apps/marauders_map/blueprint.py | New blueprint wiring the Marauder's Map stack; ReidMapModule.stop_movement output is not remapped to the teleop_active topic, so web-UI stop signals won't reach BBoxSelectionModule or TargetLockModule. |
| dimos/robot/unitree/connection.py | Adds configurable WebRTC connection method, head look-up on stop, and two-step FreeWalk+SwitchJoystick; _cancel_stop_timer now calls look_up() which can fire spuriously during mid-movement timer cancellation. |
| dimos/robot/unitree/go2/connection.py | Adds FreeWalk initialization (double-publish + SwitchJoystick) and a daemon heartbeat thread that re-asserts FreeWalk every 15 s; heartbeat lacks a clean shutdown signal. |
| dimos/robot/custom/modules/bbox_selection_module.py | New BBoxSelectionModule: click-to-bbox selection with RLock protection, stop_movement and clear_selection_request inputs; logic is sound and well-tested. |
| dimos/robot/custom/tasks/bbox_distance_behavior_module.py | New BBoxDistanceBehaviorModule: proportional distance-follow control loop; implementation is straightforward with proper RLock usage and stop-event for the background command thread. |
| dimos/navigation/movement_manager/movement_manager.py | Adds _is_navigation_click filter to ignore color_image-frame clicks in the movement manager, and publishes stop_movement before handing a new goal to the planner; logic is correct. |
| dimos/visualization/rerun/websocket_server.py | Refactors click/twist field extraction to use a _coerce_float helper with null-safe fallback and adds verbose click logging; straightforward improvement with no issues. |
| dimos/robot/unitree/keyboard_teleop.py | Fixes macOS crash by using setdefault("SDL_VIDEODRIVER", "cocoa") on darwin instead of unconditionally forcing x11; correct and safe. |
| scripts/run-blueprint.sh | New helper script that runs any blueprint and tee-logs output to /tmp/dimos-logs/ with auto-rotation; handles SIM/REPLAY/ROBOT_IP modes and the mjpython dispatch for MuJoCo on macOS. |
Sequence Diagram
sequenceDiagram
participant Browser as Browser (port 7782)
participant ReidMap as ReidMapModule
participant BBoxSel as BBoxSelectionModule
participant TgtLock as TargetLockModule
participant BBoxDist as BBoxDistanceBehaviorModule
participant MovMgr as MovementManager
participant Robot as Go2 Robot
Browser->>ReidMap: "Socket.IO select {id}"
ReidMap->>BBoxSel: stop_movement Out[Bool] NOT CONNECTED missing remap
ReidMap->>BBoxSel: clicked_point PointStamped bbox center
BBoxSel->>TgtLock: selected_bbox Detection2DArray
TgtLock->>BBoxDist: locked_bbox Detection2DArray
BBoxDist->>MovMgr: nav_cmd_vel Twist
MovMgr->>Robot: cmd_vel
Browser->>ReidMap: "Socket.IO teleop {linear_x, angular_z}"
ReidMap->>MovMgr: tele_cmd_vel Twist
MovMgr->>BBoxSel: teleop_active Bool clears selection
MovMgr->>Robot: cmd_vel priority teleop
Browser->>ReidMap: "Socket.IO navigate {wx, wy}"
ReidMap->>MovMgr: goal_request PoseStamped
MovMgr->>Robot: "nav_cmd_vel via A*"
Reviews (1): Last reviewed commit: "demo blueprint yoloe-spatial-standoff-fo..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| @self.sio.event # type: ignore[untyped-decorator] | ||
| async def connect(sid, environ) -> None: # type: ignore[no-untyped-def] | ||
| logger.info(f"Marauder's Map client connected: {sid}") | ||
| # Send the cached map + current people so a fresh client is not blank. | ||
| if self._map_meta is not None: | ||
| await self.sio.emit("map", self._map_meta, room=sid) # type: ignore[union-attr] | ||
| await self.sio.emit("people", {"people": self._people_payload()}, room=sid) # type: ignore[union-attr] |
There was a problem hiding this comment.
_people iterated from asyncio thread while stream thread mutates it
connect calls self._people_payload() from the Socket.IO / asyncio event loop, which iterates self._people (line 741 for tid, e in self._people.items()). Meanwhile _on_detections — running on the RxPY stream subscription thread — can write to self._people (via _write_entry and _expire_people) at any time. If the stream thread adds or removes a key while the asyncio thread is mid-iteration Python raises RuntimeError: dictionary changed size during iteration, crashing the connect handler silently. Unlike _latest_image / _latest_pc, _people and _stable_to_raw_det are never guarded by self._lock when read from the asyncio thread.
| # Server | ||
| # ------------------------------------------------------------------ # | ||
| def _create_server(self) -> None: | ||
| self.sio = socketio.AsyncServer(async_mode="asgi", cors_allowed_origins="*") |
There was a problem hiding this comment.
cors_allowed_origins="*" makes the Socket.IO endpoint reachable from any web origin. On a shared or conference Wi-Fi network (common at hackathons and field demos) a page on any other device can drive the robot's selection, teleop, and navigation streams. Restricting to localhost or the field subnet provides a simple first layer of defence.
| self.sio = socketio.AsyncServer(async_mode="asgi", cors_allowed_origins="*") | |
| self.sio = socketio.AsyncServer(async_mode="asgi", cors_allowed_origins="*") # TODO: restrict to trusted origin(s) before production use |
| .remappings( | ||
| [ | ||
| # ─── Selection chain ──────────────────────────────────────────── | ||
| # Both Rerun camera click and our web "click" feed BBoxSelectionModule | ||
| # through its existing `clicked_point` input. BBoxSelectionModule is | ||
| # the sole writer of /user_selected_bbox; TargetLockModule consumes | ||
| # it; the locked output is remapped to /selected_bbox for the | ||
| # follower task and the Rerun green-box overlay. | ||
| (BBoxSelectionModule, "selected_bbox", "user_selected_bbox"), | ||
| (TargetLockModule, "selected_bbox", "user_selected_bbox"), | ||
| (TargetLockModule, "locked_bbox", "selected_bbox"), | ||
| # ─── Follow task → cmd_vel mux ────────────────────────────────── | ||
| (BBoxDistanceBehaviorModule, "cmd_vel", "nav_cmd_vel"), | ||
| # ─── Web teleop → mux (priority) ──────────────────────────────── | ||
| (ReidMapModule, "cmd_vel", "tele_cmd_vel"), | ||
| # ─── Teleop activity → pause follow + clear selection ─────────── | ||
| # jamjam e9a82939: BBoxSelectionModule and TargetLockModule both gained | ||
| # `stop_movement: In[Bool]` so any source of teleop-activity wipes the | ||
| # current target and the old bbox can't auto-restart the follow on the | ||
| # next detection frame. | ||
| (MovementManager, "stop_movement", "teleop_active"), | ||
| (BBoxSelectionModule, "stop_movement", "teleop_active"), | ||
| (TargetLockModule, "stop_movement", "teleop_active"), | ||
| # ─── Marauder's Map world localization ────────────────────────── | ||
| (ReidMapModule, "pointcloud", "global_map"), | ||
| ] |
There was a problem hiding this comment.
ReidMapModule.stop_movement output is never remapped to teleop_active
The blueprint renames BBoxSelectionModule.stop_movement, TargetLockModule.stop_movement, and MovementManager.stop_movement all to the topic teleop_active, but ReidMapModule.stop_movement: Out[Bool] is left on the default stop_movement topic. These two topics won't be connected by autoconnect, so the stop_movement.publish(Bool(data=True)) calls inside the select, deselect, and navigate Socket.IO handlers never reach BBoxSelectionModule or TargetLockModule. When the tracker has no live detections the backup clear-via-click path also short-circuits, leaving a stale selection active that can restart following the moment a new detection frame arrives.
| def _freewalk_heartbeat(self) -> None: | ||
| """Re-assert FreeWalk locomotion every 15 s as a soft watchdog. | ||
|
|
||
| On real hardware we sometimes see the dog silently fall back to | ||
| BalanceStand (joystick lx/ly -> body lean, no walking). Re-issuing | ||
| FreeWalk is idempotent when already in that mode, so this is a | ||
| cheap insurance policy that fixes the most common "dog only sways" | ||
| report without forcing the user to restart the blueprint. | ||
| """ | ||
| while True: | ||
| time.sleep(15.0) | ||
| try: | ||
| self.connection.free_walk() | ||
| except Exception: | ||
| # Connection may be torn down during shutdown; ignore. | ||
| pass |
There was a problem hiding this comment.
_freewalk_heartbeat loops forever with no shutdown path
The heartbeat thread runs while True and is only a daemon=True thread — it lives for the entire process lifetime and cannot be interrupted during a clean stop() sequence. If teardown blocks on the connection or a command is dispatched mid-shutdown, the exception is silently swallowed and the loop continues. A threading.Event stop signal (checked after each time.sleep(15)) would let GO2Connection.stop() interrupt the loop quickly.
| if self.stop_timer: | ||
| self.stop_timer.cancel() | ||
| self.stop_timer = None | ||
| # Reaching the timeout means commands stopped arriving while moving: | ||
| # treat that as a stop and raise the head once. | ||
| if self.look_up_on_stop and self._was_moving: | ||
| self._was_moving = False | ||
| self.look_up() | ||
|
|
||
| def disconnect(self) -> None: | ||
| """Disconnect from the robot and clean up resources.""" |
There was a problem hiding this comment.
look_up() fires spuriously when _cancel_stop_timer is used to cancel (not fire) a pending timer
If _cancel_stop_timer is called while a new movement command is being prepared (to cancel the previous stop timer), self._was_moving is still True from the prior motion, so look_up() is called — tilting the head right before the robot resumes walking. _update_head_posture would then immediately call reset_posture(), causing a brief involuntary head-tilt on every command boundary. Limiting the look_up branch to the actual timeout path (e.g., by using a dedicated timer-callback method rather than the shared cancel helper) would avoid this.
3 participants
Summary
Build something you would like to use everyday!
This PR is based on the
jamjambranch.This time, we used DimOS to build a focus helper: the robot dog helps keep your phone away from you during a specific period of time, so you can stay focused.
Video link: Click here
Deck link: Click here
Thanks to our lovely designer.
This was our original idea.
What We Implemented
What We Did Not Implement
On macOS, DimOS MuJoCo also calls OpenCV. When our custom module needs to call OpenCV too, it can cause a multithreading conflict and crash, so we could not finish continuous tracking in time.
What We Love
We really love the design of
ModuleandBlueprint. They can be freely combined, just like LEGO bricks.Some people on our team work on BSP and a little bit of circuit design, and they also really liked this design. They said it feels very intuitive.